Augmented Proximal Policy Optimization for Safe Reinforcement Learning

Safe reinforcement learning considers practical scenarios that maximize the return while satisfying safety constraints. Current algorithms, which suffer from training oscillations or approximation errors, still struggle to update policy efficiently with precise constraint satisfaction. In this article, we propose Augmented Proximal Policy Optimization (APPO), augments Lagrangian function of primal constrained problem via attaching a quadratic deviation term. The constructed multiplier-penalty dampens cost oscillation for stable convergence being equivalent precisely control costs. APPO alternately updates and multiplier solving augmented primal-dual problem, can be easily implemented by any first-order optimizer. We apply our methods in diverse safety-constrained tasks, setting new state art compared comprehensive list safe RL baselines. Extensive experiments verify merits method easy implementation, convergence, control.